{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Taming time series\n",
    "\n",
    "Time series are collections of points collected at successive times, usually with equal time intervals between them.\n",
    "\n",
    "## Importing and visualising Time Series data\n",
    "\n",
    "In this module, we will consider the \"bikes\" dataset again but this time taking a time-series perspective. \n",
    "\n",
    "* Load the data `bikes.csv`\n",
    "* Specify that you want to parse the column `'dates'` as dates using `parse_dates=['dates']` and\n",
    "* Use the dates as the index column\n",
    "\n",
    "Check with `.head()` that everything is fine then\n",
    "* Display the temperature time series (you can either use `pandas` plotting facility by doing `bikes.plot(...)` or use `matplotlib` directly)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bikes = pd.read_csv('data/bikes.csv', parse_dates=['date'], index_col='date')\n",
    "bikes.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bikes.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise: only plot the temperature\n",
    "With the graph above it's hard to get an indication for the temperature in the data. Plot a graph with just the temperature as y-axis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bikes.plot(y='temperature', figsize=(12, 3), fontsize=12)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise: Number of bikes in a given month\n",
    "\n",
    "From the chart above, you can see that the data covers two years and that there is a similar pattern in both years which is to be expected for the temperatures. \n",
    "\n",
    "The data can be queried according to dates. For example, you can aggregate the data from just January 2012 and check the sum of bikes hired that month. Can you compare that with the number of bikes hired in August? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "jan_start = pd.Timestamp(\"1st January 2012\")\n",
    "jan_end = pd.Timestamp(\"31st January 2012\")\n",
    "bikes_jan = bikes[jan_start:jan_end]['count'].sum()\n",
    "\n",
    "aug_start = pd.Timestamp(2012, 8, 1)\n",
    "aug_end = pd.Timestamp(2012, 8, 31)\n",
    "bikes_aug = bikes[aug_start:aug_end]['count'].sum()\n",
    "\n",
    "print(\"{0:.0f} bikes in January vs {1:.0f} bikes in August.\".format(bikes_jan, bikes_aug))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Resampling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can aggregate time series by resampling the points on a coarser time level. \n",
    "\n",
    "* Use the `.resample` to get the data corresponding to monthly averages\n",
    "* Display the `temperature` time series for the monthly averages. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'bikes' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-1-59360e1bc45e>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mbikes_monthly\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mbikes\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mresample\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'M'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmean\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      2\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0mplt\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfigure\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfigsize\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m12\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m3\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0mplt\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mplot\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mbikes_monthly\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtemperature\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"-o\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mNameError\u001b[0m: name 'bikes' is not defined"
     ]
    }
   ],
   "source": [
    "bikes_monthly = bikes.resample('M').mean()\n",
    "\n",
    "plt.figure(figsize=(12, 3))\n",
    "plt.plot(bikes_monthly.temperature, \"-o\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Resample by the mean of each week and uses the humidity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bikes.plot(y='humidity', figsize=(12, 3), fontsize=12)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bikes_weeks = bikes.resample('W').mean()\n",
    "\n",
    "plt.figure(figsize=(12, 3))\n",
    "plt.plot(bikes_weeks.humidity, \"-o\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Parsing custom date formats\n",
    "\n",
    "When you loaded the bikes dataset, Pandas automatically detected the format of the dates for you.\n",
    "This might often \"just work\" but there often will be cases where you need to be careful about parsing and might have to do it yourself.\n",
    "\n",
    "Load the data `NZAlcoholConsumption` and have a look at it without specifying a column to parse for dates. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "alcohol_consumption = pd.read_csv('data/NZAlcoholConsumption.csv')\n",
    "alcohol_consumption.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This dataset contains data aggregated by quarters, the timestamp is formatted in a string where the first 4 characters represent the year and the last two the quarter. \n",
    "To transform the timestamps in dates that pandas can directly use, you can write a parser function. \n",
    "\n",
    "\n",
    "### Exercise: parsing quarter\n",
    "Write a function `parse_quarter` that takes a string of the form `YYYYQN` and convert it to `pandas.Timestamp` object. Use the following conversion for the quarters:\n",
    "\n",
    "* Q1 --> mar 31\n",
    "* Q2 --> jun 30\n",
    "* Q3 --> sep 20\n",
    "* Q4 --> dec 31"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "def parse_quarter(string):\n",
    "    \"\"\"\n",
    "    Converts a string from the format YYYYQN in datetime object at the end of quarter N.\n",
    "    \"\"\"\n",
    "    \n",
    "    # Note: you could also just retrieve the first four elements of the string\n",
    "    # and the last one... Regex is fun but often not necessary\n",
    "    year, qn = re.search(r'^(20[0-9][0-9])(Q[1-4])$', string).group(1, 2)\n",
    "    \n",
    "    # year and qn will be strings, pd.datetime expects integers.\n",
    "    year = int(year)\n",
    "    \n",
    "    date = None\n",
    "    \n",
    "    if qn=='Q1':\n",
    "        date = pd.Timestamp(year, 3, 31)\n",
    "    elif qn=='Q2':\n",
    "        date = pd.Timestamp(year, 6, 30)\n",
    "    elif qn=='Q3':\n",
    "        date = pd.Timestamp(year, 9, 20)\n",
    "    else:\n",
    "        date = pd.Timestamp(year, 12, 31)\n",
    "        \n",
    "    return date\n",
    "# Check that it works!\n",
    "print(parse_quarter(\"2000Q3\")) # should show 2000-09-20 00:00:00"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Giving the parser to pandas\n",
    "\n",
    "Pandas can parse dates using a custom made parser such as the one you just defined. For this just specify your function in the `date_parser` option."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# reload the data using your parser, set the index to the date \n",
    "alcohol_consumption = pd.read_csv('data/NZAlcoholConsumption.csv', \n",
    "                                  parse_dates=['DATE'], \n",
    "                                  date_parser=parse_quarter,\n",
    "                                  index_col='DATE')\n",
    "alcohol_consumption.sort_index(inplace=True)\n",
    "alcohol_consumption.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise: Display the time series\n",
    "\n",
    "Now, have a look at the consumtion of wine and beer, show both on the same figure. Discuss the two time series."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(8, 6))\n",
    "plt.plot(alcohol_consumption.TotalWine, \n",
    "         '-o', label='Wine')\n",
    "plt.plot(alcohol_consumption.TotalBeer, \n",
    "         '-o', label='Beer')\n",
    "plt.legend(fontsize=12)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The plots show that the two time series have similar patterns in terms of seasonality but different trends.\n",
    "Both show that alcohol consumption is maximum in the last quarter of the year and is usually at its lowest in the second quarter. \n",
    "The average beer consumption seems stable during the years, while the wine consumption seems to be steadily increasing. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise: resample the data per year (12 months) \n",
    "Can you resample the data per year (12 months) and see whether the trends come out better? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "alc_yearly = alcohol_consumption.resample('12M').mean()\n",
    "\n",
    "plt.figure(figsize=(12, 3))\n",
    "plt.plot(alc_yearly.TotalWine, \"-o\", label=\"Wine\")\n",
    "plt.plot(alc_yearly.TotalBeer, \"-o\", label=\"Beer\")\n",
    "plt.legend(fontsize=12)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Moving Windows\n",
    "\n",
    "In the cells below you will explore the effect of applying a \"Rolling Average\" to the data i.e.: look at a number of successive points, take the average, and replace the window by the average (either at the extreme right of the window, or at the center)\n",
    "\n",
    "* Use the `rolling` method from `pd.Series` ([documentation](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.rolling.html#pandas.Series.rolling))\n",
    "* specify a window of 4 points\n",
    "\n",
    "plot the averaged line and the original time series and discuss. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(8, 6))\n",
    "plt.plot(alcohol_consumption.TotalWine,\n",
    "         '-o', label='wine consumption')\n",
    "rolling_mean = alcohol_consumption.TotalWine.rolling(window=4).mean()\n",
    "plt.plot(rolling_mean, label='trend')\n",
    "plt.legend(fontsize=12);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The rolling mean curve seems to capture the trend nicely and removes much of the seasonal movements. \n",
    "This curves allows to better appreciate the overall increase of wine consumption over time as well as the dip in consumption in 2008. \n",
    "\n",
    "To explore this rolling average further, it's nice to look at widgets. Have alook at the cell below and modify at will. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from ipywidgets import interact, interactive, fixed, interact_manual\n",
    "import ipywidgets as widgets\n",
    "\n",
    "def rolling_avg_plot(window_size):\n",
    "    plt.plot(alcohol_consumption.TotalWine, \n",
    "             '-o', label='wine consumption')\n",
    "    rolling = alcohol_consumption.TotalWine.rolling(window=window_size).mean()\n",
    "    plt.plot(rolling, label='trend')\n",
    "    plt.legend();\n",
    "    plt.show()\n",
    "\n",
    "interact(rolling_avg_plot, window_size=(0, 10));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise: plot the moving sum with a window of width 4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(8, 6))\n",
    "plt.plot(alcohol_consumption.TotalWine,\n",
    "         '-o', label='wine consumption')\n",
    "rolling_mean = alcohol_consumption.TotalWine.rolling(window=4).sum()\n",
    "plt.plot(rolling_mean, label='trend')\n",
    "plt.legend(fontsize=12);\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Differencing\n",
    "\n",
    "Differencing amounts to looking at the time series formed of differences between values separated by a given lag: \n",
    "\n",
    "$y'_t = y_t-y_{t-1}$\n",
    "\n",
    "for a lag of 1. Show the time series for `TotalWine` and the differenced one (with lag 1). What do you observe? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(8, 6))\n",
    "plt.plot(alcohol_consumption.TotalWine, '-o', \n",
    "         label=\"original ts\")\n",
    "plt.plot(alcohol_consumption.TotalWine.diff(1), '-o', \n",
    "         label=\"differenced ts (lag=1)\")\n",
    "plt.legend(fontsize=12)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To get a feel for what a good lag should be (though here, intuitively, you should realise that a lag of `4` is a good idea), you can look at the cell below that shows differenced series for increasing lags. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def differencing_plot(d):\n",
    "    differenced_ts = alcohol_consumption.TotalWine.diff(d)\n",
    "    plt.plot(differenced_ts, '-o')\n",
    "    plt.show()\n",
    "\n",
    "interact(differencing_plot, d=(1, 10));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Autocorrelation\n",
    "\n",
    "Autocorrelation measures the correlation (similarity) between the time series and a lagged version of itself. \n",
    "\n",
    "* Use the `autocorr` method ([documentation](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.autocorr.html)) to compute the autocorrelation for lag from 1 to 13\n",
    "* Display the values of the autocorrelation using a stem plot (`plt.stem`) \n",
    "\n",
    "What do you observe? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lags = range(1, 13)\n",
    "autocorrs = [alcohol_consumption.TotalWine.autocorr(lag=lag) \n",
    "                   for lag in lags]\n",
    "plt.figure(figsize=(8, 6))\n",
    "plt.stem(lags, autocorrs)\n",
    "plt.xlabel(\"Lag\", fontsize=12)\n",
    "plt.ylabel(\"Autocorrelation\", fontsize=12)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's quite clear from this plot that the time series is self-similar to itself with a lag of 4 and consistently so (so also with a lag of 8, 12, etc)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
